Welcome to buy time in three hours.
Hello, everyone, this is Lecturer eight, data processing and calibration in machine learning.
By the end of this lecture, you will be able to calibrate the data for machine learning and process
the data for regulation and classification problems.
Calibration is necessary for machine learning because when we see in the next lecture some activation
functions in machine learning or neural networks, they're going to comprehend that having data that
each attribute of aid over various data points has values between different ranges of magnitudes would
harm the way that machine learning learns from the training data.
So one solution is to calibrate our data or each column of data, I mean, each attribute or each characteristics
across multiple data points between certain lower bound and upper bound, like an 011 or minus one and
one.
Another way is called normalization.
Normalization is to transform each column of data or each attribute across the data points, such the
standard deviation becomes one and the average values becomes zero.
But in this lecture, we only practice calibration at the beginning.
We have to import some necessary libraries.
As usual, we have Ponder's and non-pay Dennen from scalar.
We are importing our preprocessing as PJI and then from cross we are importing utils as queue, cross
or tenso for a cross is one of the famous popular and powerful libraries with various machine learning
tools.
So let's run this cell for the time being.
So we first load some examples from sample data directory.
We are basically importing the train and testator corresponding to California housing using read underlying
CSU function of pundits and we're going to look at the top five rows or top five data points of our
training data.
So let's run this.
So like I explained at the beginning, each column of data has its own range.
For example, the range of line detailed in this particular example are negative values.
Latitudes are in a scale of thirty four, thirty five, thirty three.
But in the other hand, our housing values are in scale of thousands or the total number of bedrooms
are in scale of hundreds.
So we have to bring the data into a balance range of values.
Our machine learning model would act much better on such data.
We need a balanced range of values across different characteristics or attributes.
So in order to do that, we perform calibration and calibrating.
We transform every column of data between a predefined lower bound and upper bound like zero and one
or minus one and one.
And we do this for all the columns.
OK, let's talk about data processing for regression problems.
So we first load some data and then in this particular example, our training and testing data have
already been separated into two CSP funds.
In this particular example, like we have seen before, the last column of our data, our output data,
so we can get our data input of training and data output of training by getting whatever that is from
column number zero to column number minus two, but minus one.
And for our output is just the last column and we can call it using indexing minus one.
Similarly for our testing data.
So just Rudisill Denay.
So I basically show you how we calibrate our data so that PJI function from across that utils has a
function called min max scale.
If we're going to actually create a function or a model function using that minimizes the scalar and
within the practices we are defined one of the variables that is feature range.
And here we define our lamon an upper bound in a form of a tupa.
So in this case, our lower bound is an upper bound is one.
And this function is just for calibrating our input data.
We don't see any information about input data here, but we have it actually here.
So once you have that function or model, define Opta, then here we can fit that function or model
over our training data.
Not that we assume we put aside our testing data and we have no information regarding them until the
exam date, until we have already been trained by the training data.
So whatever information we have currently, our training data.
So this mean, Max, a scalar kind of reserve?
The actual original ranges of different kinds of data or attributes in our inputs are training, and
then later, when we actually use this function for our testing data, our testing data might fall beyond
the range of 011.
And there's a chance for that.
Although if our whole repository is a good representation of the problem, then that doesn't happen
frequently.
But there is a probability.
So you do this for the inputs data and you feed that function for calibration using the input of the
training data.
You have a similar function defined for your output.
And here I am fitting it with the output of training.
So you have a fun calibration for input and a fine calibration for output.
Then here are calibrated training.
Data is simply defined calibration for the input that you use, the function, transform and then put
your data in training and whatever you see here as the data in training calibrated, all of the columns
will be automatically transformed between zero and one.
So the maximum value of each column here is one and the minimum value of each column here is zero for
sure, because we already fitted our model using the input of training data here and the output of training
data here.
But when we want to compute the calibrated inputs of testing or the calibrated output of the testing,
like I said before, there is a chance that they calibrated version far beyond the range of 011 for
some of our testing data points.
And the reason is that maybe there has been some values in our testing data point that is beyond the
original maximum minimum values of the corresponding attribute in our original training data.
So in that case, we expect that our testing calibrated data for that particular for those particular
data points fall beyond the range of 011.
But that shouldn't be too many data points and it would be OK.
So we have the same story for our outputs of training and testing, and now we have our training calibrated
and testing calibrated data.
So I just want this.
And here I'm going to show you guys the data in training calibrated.
And if you see that all the values here are between zero and one.
And if I just show you, for example, the first column that marks the maximum value is one, the minimum
value is zero.
And this is the same story for all the columns.
Let me tell.
It is so.
OK, our calibration process for both inputs and outputs in regulation problems are very similar because
both outputs and inputs, they are usually real values.
But for classification problems, that's a little bit different.
In this example, we are using the amnesty to amnesty.
Data is also available in the sample folder here for you guys.
So what is amnesty to amnesty to?
Is the standard data benchmark very popular one and each data point in it represent a handwritten digit
data.
It's basically handwritten, but we turn the image into pixel intensities and each pixel has a value
between one and two hundred and fifty five.
And each of these images you see here is basically twenty eight by twenty pixel.
Or if you show that twenty eight by 20 pixel intensities in a form of a data point, that's going to
be twenty eight times twenty eight equals to seven hundred and eighty four pixel intensities.
So each of your data points is like seven hundred and eighty four values of pixel intensities.
There are nine handwritten digits from zero to nine and for each one of them are a good number of examples
of handwritten digits by different people.
So the goal here is that we give the machine learning algorithm any of these pixel intensities, and
we expect the machine learning to estimate the corresponding class of that particular and return data
point of pixel intensities.
You can Google any slate and read about this data repository further here.
We're going to load our misclassification data and we're going to split inputs and outputs.
So here from the sample data folder, I'm reading the CSP for data, for the training data and the CSP
data for the testing data.
In this particular example, our output categories zero to nine have been stored on the very first column
of our data.
If you're looking to the data training or testing, that's why our input data or the pixel intensities
or whatever that comes from column number one to the rest and column number zero is basically our outputs
and we have them for both training and testing here and here.
I'm just printing for you the shape of these training and testing inputs and outputs.
So let's just run this up and here we go.
We have about twenty thousand training data and about 10000 testing data.
So for calibration of the input data, it is very similar to what we have done.
What a regression problem.
It is actually exactly the same.
And now we can do the same thing that we have done for the regression to get our data input training
calibrated and data input testing category.
But as you see, I commented the ones we had for our outputs.
And the reason is that we don't need to calibrate the output because our outputs and classification
problems are categories.
But in this particular example, we need to turn these categories into binary categories or what?
In that case, you got cross utils method or class tells you as to categorical.
And this is like if you have one category to be category number two, like this one, then we assume
that we have 10 binary values and the value corresponding to the third binary beat is going to be one
and there is going to be zero.
It means that the value corresponding to column number two, which is the third column, are going to
be one and there is going to be zero.
So this can you the two categorical or training output and testing output are going to turn our outputs
into the categorical or binary format.
This one is sound.
As you see, the shape of my outputs are now about ten thousand and ten because for each data point
we have ten binary outputs.
So let's just look at one of them here in the NEWSROOM.
I'm going to print it here.
I'm going to say, for example, the first one, all of them.
And here we go.
All of the values are zero except this one, which means that this data point belongs to category number
zero.
One, two, three, four, category number five.
Let me with this.
So that's it.
Our data, whether regulation or classification have been split into training and testing calibrated
or turn into categorical or binary version, and then they are ready to be inputted into our machine
learning techniques that we are going to talk about neural network in the next session.
And we're going to see an example of how to implement this data for training and testing our machine
learning techniques.
In this lecture, you learn about how to calibrate the data for machine learning and how to process
the data for regression and classification problems.
In the next lecture, we will go over a brief introduction to neural networks.
Thank you for your attention.
See you all in the next lecture.